Adding On-demand Training Data Notebook #162
Conversation
KennSmithDS
changed the title
There was a problem hiding this comment.
Thanks, I left a few comments inline. Still making my way through the example.
One general comment: I'm pretty uncomfortable having moderately complex code in this examples. I'd much prefer that things like temporal_buffer, mind_cloud_cover_scene and even get_landsat_8_match be generalized and put into a dedicated library, where it can be properly unit tested. With it here in a notebook it's not easy to test and not easy to reuse.
| "source": [ | ||
| "Once you have your API key, you will need to create a default profile by setting up a .mlhub/profiles file in your home directory. You can use the `mlhub configure` command line tool to do this:\n", | ||
| "\n", | ||
| "`$ mlhub configure`<br>\n", |
There was a problem hiding this comment.
!mlhub configure --api-key={MLHUB_API_KEY}
as a regular code cell should work.
| "id": "2ba7d7b2-7fed-4412-8c9b-448baad6e595", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "This helper function below encapsulates the process of querying a STAC API endpoint to fetch an ItemCollection matching query criteria." |
There was a problem hiding this comment.
IMO this helper function isn't adding much value over just using catalog.search directly. I'd rather teach users how to use catalog.search.
Can you remove uses of search_stac_api and the client_catalog.search instead?
| "id": "2e131160-50bb-487f-8fcb-96d37ce80167", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "We could certainly use the method above to query label Items directly from our connection to the Radiant MLHub API endpoint. However, on very large collections, such as in the case with BigEarthNet, pagination becomes a bottleneck issue in obtaining and resolving STAC items, as it only returns 100 items at a time. Querying the entire Collection of nearly ~600,000 Items could take hours.\n", |
There was a problem hiding this comment.
FWIW, the limit argument in pystac-client controls the size of the pages. But agreed that fetching 600,000 items through an API isn't what we should recommend.
| "if not os.path.exists(label_collection_path):\n", | ||
| " collection = Collection.fetch(BIGEARTHNET_LABEL_COLLECTION)\n", | ||
| " archive_path = collection.download(TMP_DIR)\n", | ||
| " !tar -xf {archive_path.as_posix()} -C {TMP_DIR}\n", |
There was a problem hiding this comment.
It's unfortunate that this decompression takes so long :/ Any thoughts on if you can operate directly on the compressed .gz file? Probably not.
There was a problem hiding this comment.
I agree, it is unfortunate how long the decompression takes.
I didn't have a great work around to this, as it is related to your previous comment about using limit, and taking a random sample of Items from the larger dataset of 600,000 Items.
I suppose if we fetch only a few thousand to begin with, pagination shouldn't be an issue then we don't need to deal with the .tar.gz file. However that likely won't be a random result, I'm guessing the API will just return the first XXX Items?
| "id": "e419d96f-f8b9-4911-b5d1-4a4077767062", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "If we had the source collection archive downloaded and uncompressed in the same parent directory as the labels collection, we could reference the source Items and images directly. However the BigEarthNet source collection is over 60GB when compressed. Therefore to work around the disk size limitations of a Planetary Computer instance, we can query the same source items from the MLHub API endpoint, the same way we got the labels, but filter to the exact source item using IDs." |
There was a problem hiding this comment.
What's your preferred workflow as a user, and from MLHub's point of view? Do you want people making local copies of this dataset, or do you want them fetching from your storage on demand?
IMO, the ideal workflow is on-demand fetching from blob storage in the same region + caching, but I'm curious what you think.
There was a problem hiding this comment.
In my short time here at Radiant, I've seen a divergence between what is a preferred workflow for our users vs. the workflow most of our data users/consumers follow.
Personally I agree that the ideal workflow is an on-demand fetching of assets from blob storage via traversing a catalog/STAC API on a VM or notebook server in the same region. However, my assumption is that a majority of our users are downloading the datasets directly to their local computers so they can do machine learning, or other geospatial analytics on them, or many aren't familiar enough with STAC/PySTAC and STAC APIs to fetch the data that way.
Also with any on-demand hosted notebook server environment comes the issue of limited persistent volume sizes, especially if folks are accustomed to downloading instead of fetching and caching without writing to disk.
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "if best_l8_match:\n", |
There was a problem hiding this comment.
Same question about potentially not having a match.
There was a problem hiding this comment.
See above for if source_items comment thread
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "explore_search_extent(ItemCollection([best_l8_match]))" |
There was a problem hiding this comment.
Can you also plot the s2 chip's bound here?
There was a problem hiding this comment.
Similar to other comment threads, would it be best to strip this out of the helper function, and just have a few cells of repeated code for exploring the API search results?
There was a problem hiding this comment.
Clarifying question on this, do you mean the overall bounds of the ItemCollection returned (e.g. minX, minY, maxX, maxY)? Each Item returned from the API search will have it's own bounding box.
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "client = dd_client(\n", |
There was a problem hiding this comment.
Just use dask.distributed.Client() (or a GatewayCluster & get_client if doing this on a distributed cluster)
There was a problem hiding this comment.
for both ps_client and dd_client I simplified by removing the import alias, so users will directly use distributed.Client and pystac_client.Client
| "outputs": [], | ||
| "source": [ | ||
| "# this cell will only work on PC or a machine with gateway cluster configured\n", | ||
| "# gateway = dask_gateway.Gateway()\n", |
There was a problem hiding this comment.
IMO: pick whether or not you want this example to run on a cluster. If it runs without a cluster, then we can just remove this.
| " Landsat 8 DataArray that has been cropped to label bbox\n", | ||
| " \"\"\"\n", | ||
| " # read label Item object\n", | ||
| " label_item = Item.from_file(\n", |
There was a problem hiding this comment.
This, and the other functions touch the local filesystem won't work with a distributed cluster. They'd only work with a LocalCluster.
If we're mentioning possibly using a distributed cluster then you'd need to restructure this. Most likely, you'd need to store everything to Azure Blob Storage and use it as a kind of shared file system that each worker could read / write to.
There was a problem hiding this comment.
For purposes of this tutorial, seems to make sense to focus on the workflow of gathering and processing the Landsat 8 data off of Planetary Computer with a LocalCluster, and not convoluting that with performing this workflow on a distributed cluster.
|
@TomAugspurger do you know why the For example in this block of code, I'm fetching the Sentinel-2 source imagery from the Azure Blob storage for our MLHub. We know the chips all to be 120x120 pixels, but the stack object dimensions vary from 122x122 up to 130x130.
P.S. sorry I don't know how you're doing the cool Jupyter Notebook integration. |
|
Closing in favor of #171 due to rebasing issue |
PR to merge the notebook tutorial for creating on-demand training data from the Planetary Computer data catalog when starting from a Radiant MLHub dataset.